Modeling Local Interest Points for Semantic Detection and Video Search at TRECVID 2006

Local interest points (LIPs) and their features have been shown to obtain surprisingly good results in object detection and recognition. Its effectiveness and scalability, however, have not been seriously addressed in large-scale multimedia database, for instance TRECVID benchmark. The goal of our works is to investigate the role and performance of LIPs, when coupling with multi-modality features, for high-level feature extraction and automatic video search. In high-level feature extraction, we explore LIPs with both local description and spatial distribution for characterizing and sketching semantic concepts respectively. Two visual dictionaries, based upon universal visual keywords and concept-based visual keywords, are generated for experiments. The 39 concepts are learnt by SVM in vector space model with the support of both dictionaries. In addition, the distribution of LIPs is also exploited for detection with the multi-resolution and embedded Earth Mover’s Distance settings. We submit six runs by incorporating the two properties of LIPs with other modalities including grid-based color moment and wavelet texture. CityU-HK1: average fusion of 4 SVM classifiers using universal visual keywords, distribution of LIPs, grid based color moment, and wavelet texture. CityU-HK2: average fusion of 3 SVM classifiers using universal visual keywords, grid based color moment, and wavelet texture. CityU-HK3: average fusion of 3 SVM classifiers using distribution of LIPs, grid based color moment, and wavelet texture. CityU-HK4: grid based apriori mining method. CityU-HK5: average fusion of 3 SVM classifiers using concept-based visual keywords, grid based color moment, and wavelet texture. CityU-HK6: baseline method by average fusion of 2 SVM classifiers using grid based color moment, and wavelet texture. Results show that the LIP-based features could generate comparable results with traditional color/texture features. By incorporating the LIP-based features upon color moment and wavelet texture, an improvement of 51.4% is reported. In automatic search, we study the performance of query-by-example (QBE) and mini-ontology (39 concepts) on top of baseline text search. In QBE, the properties of LIPs are utilized as one of features for retrieval. In mini-ontology, we measure the similarity of query terms to 39 concepts and adopt various heuristic settings (Detailed in Section 3.4) to test its significance for search. We submit six runs, for all queries, to show the advantage of search with the mini-ontology as semantic filters, and compare its performance to the classical text search. CityU-HK1: multimodal automatic run using text search, mini-ontology with setting 1. CityU-HK2: multimodal automatic run using text search, mini-ontology with setting 3. CityU-HK3: multimodal automatic run using text search, QBE, and mini-ontology with setting 1. CityU-HK4: multimodal automatic run using text search, mini-ontology with setting 4. CityU-HK5: multimodal automatic run using text search, mini-ontology with setting 3. CityU-HK6: required baseline run using ASR/MT transcripts only.